Universal method and apparatus for mutual sound and light correlation

ABSTRACT

Method and apparatus utilizing the method for continuous correlation of sound to light and light to sound, and more particularly a method based on a universal equation such that it can be applied to all wavelengths of light and sound seamlessly without any intuition or subjective inputs. Within this method, the entire audible sound spectrum mutually correlates to the entire visible light spectrum. A light and sound correlation apparatus disclosed is an apparatus where the wavelength of light and/or sound is input and the correlating sound and/or light is output. This invention is revealed in a plurality of embodiments for human needs, such as providing an interactive color and sound correlation display apparatus and slide rule, a color MIDI file generating apparatus, a colorized visual orchestration apparatus, an art composer apparatus, spatial information apparatus with color and sound, and a color language apparatus.

TECHNICAL FIELD

The present invention relates to a method, and an apparatus exploitingsaid method, for continuous correlation of sound to light and light tosound.

BACKGROUND ART

Since ancient times, it has been a desire and need of humankind toassociate colors and sounds and pursue its benefits. Even today, suchefforts remain intuitive and subjective with one thing in common: theyreflect a need of humankind but are not capable of being consistentlyapplied to all colors and sounds humans can see and hear. This is aresult of the lack of science in these efforts. If a universalcorrelation had been found, then the entire audible range couldcorrelate to the visible range without any human intervention ormodification. Such a correlation must be seamless in order to have acontinuous dialogue between light and sound to address human needs.

Current continuations of intuitive and subjective efforts areexemplified in several patents, which all rely on a user dependentrelationship. At best, an association, not a correlation is maderelative to a reference sound frequency or color.

U.S. Pat. No. 6,686,529 B2 (2004) employs an equation to convert onesignal of the audible frequency into a signal of a visible frequency,which involves a reference visible frequency (Fl, fl) to be inputted bythe user and employs arbitrary color harmony schemes. In addition, themethod involves selection of a reference color from a table or a scaledegree-dividing rate and modifies frequencies to fit the audible andvisible ranges. Hence, this method proposes a user input dependent colorand sound conversion criterion rather than a universal correlationequation.

Another prior art document, namely U.S. Pat. No. 4,378,466 (1983)discloses a method for conversion of acoustic signals into visualsignals, wherein each audio frequency is assigned a respective colorhue. This method also involves an artificial assignment procedure amongcolors and sounds.

A sound-picture converter is explained in Japanese Patent 63,184,875,(1988) wherein each element picture is allowed to correspond to a tonecolor, and each element of the picture is converted to sound based onthe said correspondence. A similar converter is disclosed in anotherJapanese Patent 3,134,697 (1991).

A PCT application no. WO 81/00637 (1981) discloses a visual method ofrepresenting sound by color, consisting of a subjective division of thecolor spectrum into twelve hues and correlating each of the twelve notesof the musical octave with each hue in such a way that degrees ofconsonance and dissonance between notes are claimed to correlate withthat between the corresponding colors.

Japanese Patent 01,091,173 A2 (1989) allows MIDI (Musical InstrumentDigital Interface) signals of a music piece to be displayed on a TV(cathode ray tube) screen in terms of pictures of four basic types ofmusical instruments, i.e. piano, strings, horns, and rhythm instruments.An electronic circuit processes the MIDI signals to display thecorresponding musical instrument pictures simultaneously with the music.This patent simply allocates MIDI signals to their corresponding musicinstrument displays and does not involve a color sound correlationwhatsoever.

Japanese Patent 22,000,734 A2 (2002) describes a musical therapy supportdevice, which accompanies a screen output for the music being played byprocessing its MIDI signals. This device is not based on any scientificcolor-sound correlation.

Japanese Patent 04,170,574 A2 (1992) describes a method for playing acolor-classified instrument by a color-classified score. According tothis patent, different colors are assigned to different notes of music,and a music instrument with colored keys is played accordingly. Thecolor assignment in this patent does not involve any scientificcolor-sound correlation.

U.S. Pat. No. 6,515,210 B2 (2003) issued to Shibukawa discloses amusical score displaying apparatus and method for a keyboard with acolor monitor. On the color monitor, a prearranged color appearsindicating which key to be pressed in performing a music piece.

In U.S. Patent Application 2004/0061668 A1, Lin describes a LED (LightEmitting Diode) based lighting apparatus operated in synchronism withthe music played. This apparatus is primarily intended for entertainmentwhere LED colors and brightness change with the frequency of sound. Inthis apparatus, the LED color and brightness selection were arbitrarily.

In U.S. Patent Application 2003/0117400 A1, Steinberg et al. disclose anapparatus which utilizes a color palette to display musical notation ona color monitor or display screen with various combinations of userselected and adjusted colors, shapes, patterns etc.

In U.S. Patent Application 2004/0074376 A1, Varme discloses a system tocolorize musical scores through a septuary system of colors based on anarbitrarily selected master color matrix.

U.S. Pat. No. 5,998,720 (1999) issued to Beatty discloses a musicteaching system and method comprising at least two musical instruments.Each musical instrument has a mechanism for producing a musical notewhen the means is activated. Each such mechanism is marked by a colorcorresponding to the particular musical note produced by the mechanism.In this method, the color sound association was determined arbitrarilysimply to allow students with color-coded hand bells to followcolor-coded cards displayed by the teacher.

U.S. Pat. No. 5,931,680 (1999) issued to Semba discloses an apparatusfor displaying beat marks corresponding to the number of beats permeasure during a performance by a musical instrument karaoke apparatus.Color of each displayed beat marks change color in synchrony with thetiming of beats. The color change method does not involve a scientificcolor-sound correlation; instead, colors simply change with apredetermined direction with the tempo of music.

In U.S. Patent Application 2004/0007118 A1, Holcombe describes a methodof music notation, which assigns distinct colors to the twelve notes ofthe C major scale. Color boxes are embedded in the conventional notationsheets. In this method, the color assignments have been arbitrarilyselected.

U.S. Pat. No. 6,660,921 B2 (2003) issued to Deverich discloses a methodfor teaching stringed instrument students how to play sheet music byusing colored fingering numbers. In this method, “easily identifiable”distinct colors were arbitrarily assigned to particular notes.

The common denominator of all the above prior art is the fact that thecolor sound associations were made arbitrarily and none of them agreewith another. Furthermore, the direct correlation between thereplication pattern of octaves in music and the replication of spectralcolors with different shades was neither recognized nor fully utilized.

None of the documents in the prior art disclose a scientific and naturalcorrelation between sound and light; instead, they stem from intuitiveand artificial conversions.

On the other hand, in the human brain, the color response subfields arearranged from low frequency (red) to high frequency (violet). Similarly,each subfield in the human brain for sound responses is arranged fromlow to high frequency. This indicates that there is a naturalcorrelation between the biological sequencing of light and sound waves,which was failed to be disclosed in prior art documents. If a colorsound correlation is to be used for human oriented applications, it mustrepresent and appeal to this natural correlation. However, up to presentsuch a natural correlation was deemed impossible.

BRIEF DISCLOSURE OF THE INVENTION

The object of the present invention is to establish a direct and uniquecorrelation between the wavelengths of visible light and audible soundin a seamless, continuous, and objective manner.

Another object of the present invention is to effectively recognize andutilize the natural human color and sound cognitions.

Still another object of the present invention is to provide a naturalcorrelation between wavelengths of sound and light.

Still another object of the present invention is to develop a mutualsound and light correlation method and apparatus that is universal, i.e.that can be applied to all wavelengths of light and sound, without anyuser intervention or dependency.

The aforementioned objects are mainly achieved through a methodutilizing a continuous function, which seamlessly correlates wavelengthsof light to sound and sound to light.

According to present invention, a method based on a universal equation,which correlates sound and light waves continuously and seamlessly forthe first time, is disclosed. Within this method, the entire audiblesound range mutually correlates to the entire visible light rangewithout any human intuition, subjective inputs, or reference pointselections. This method uniquely recognizes the natural correlation inthe human brain.

The invented method is also utilized in a light and sound correlationapparatus wherein the wavelength of light and/or sound is input and thewavelength of correlating sound and/or light is output along with otherrelevant data.

A plurality of embodiments for human needs is possible thanks to thespecial attributes of the invention, e.g. human cognition oriented,continuous, and seamless. Examples to these embodiments includeinnovative solutions for the hearing or visually impaired as well asproviding creative devices in music education.

In an aspect of the invention, the method is utilized for producing a3-D color sound correlation display apparatus for illustrative andeducational purposes, which comprises layers of color chromaticitydiagrams for different perceived brightness of colors superimposed withthe continuously correlating sound data. This diagram can be in anyphysical or electronic format.

In another aspect of the invention, the method was utilized forproducing a color and sound correlation slide rule apparatus forreferral, animation, illustration, and education purposes, whichcomprises a combination of moving and stationary parts. The slide ruleapparatus audibly and/or visually shows the color and sound correlationin any input order.

In still another aspect of the invention, the method and apparatus isutilized for generating sound and light correlation of music. A newCMIDI (Color MIDI) file generating apparatus establishes a dynamiccoupling among sound and light data. Sounds of each instrument or voiceper each time increment are coupled with correlating light data. Thisfile is an array of color and sound data, which can be output to anystorage, retrieval, printing, transmitting, display, or animationdevice, depending on the desired form of output.

In still another aspect of the invention, the method and apparatus isutilized to provide a visual orchestration apparatus where the soundsproduced by instruments are dynamically displayed with colors on aninstrument layout. This layout has at least two dimensions and maps atleast one music instrument, voice, and/or sound sources. The presentinvention, which consists of a continuous correlation between sound andlight, can colorize all sounds. Hence, a visual orchestration of anycombination of instruments and/or sounds sources are possible.

In still another aspect of the invention, the method and apparatus isutilized to produce an art composer apparatus, which generatesorchestral sounds correlated with colors in a visual image. An artcomposer apparatus comprises color inputs for each incremental gridsurface area of the image. The color inputs corresponding to each gridsurface area are assigned to orchestral instruments and processed by alight and sound correlation apparatus.

In another aspect of the invention, a method and apparatus that producessound data incorporating three-dimensional spatial information of anobject is provided. The spatial information of an object is derived fromthe fact that the shades of colors change with depth. This apparatus isespecially designed for individuals with William's Syndrome who haveexceptional music ability but poor perception of depth.

In another aspect of the invention, a visual orchestration apparatus isutilized to provide a color language apparatus to aid perception ofmusic played on any AV (audio-visual) apparatus, such that in analogy tosign language, a hearing impaired can visually follow sounds in a musicperformance on a TV screen, on which a picture in picture boxdynamically displays the visual orchestration simultaneously with themain broadcast.

DETAILED DISCLOSURE OF THE INVENTION

The file of this patent contains some drawings in color. Copies of thispatent with color drawings will be provided in the national phase ifrequested.

The above object, and other features and advantages of the presentinvention will become more apparent after a reading of the followingdetailed description when taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 shows a flowchart of the present method;

FIG. 2 shows a block diagram of the present light and sound correlatingapparatus;

FIG. 3 shows a typical construction step of a color and soundcorrelation display apparatus according to the present invention;

FIG. 4 shows a color and sound correlation display apparatus accordingto the present invention;

FIG. 5 is a schematic view of the color and sound correlation slide ruleapparatus according to the present invention;

FIG. 6 is a block diagram of the color MIDI (CMIDI) file generatingapparatus according to the present invention;

FIG. 7 shows an orchestral instrument layout and the digital simulationof that layout with section numbers according to the present invention;

FIG. 8 shows a block diagram of a visual orchestration apparatusaccording to the present invention;

FIG. 9 is a table showing a sample CMIDI file for a number ofinstruments in an orchestra according to the present invention;

FIG. 10 shows a resulting view corresponding to one time frame of agenerated CMIDI file according to the present invention;

FIG. 11 shows a series of views corresponding to a series of time framesof a generated CMIDI file according to the present invention;

FIG. 12 shows a flowchart of an art composer apparatus according to thepresent invention;

FIG. 13 shows another embodiment of a present invention for a spatialinformation apparatus with color and sound for individuals withWilliam's Syndrome according to the present invention;

FIG. 14 shows a block diagram of a color language apparatus according tothe present invention.

In an aspect of this invention, there is provided a method comprising apower ratio between sound wavelengths (λ_(s)) and light wavelengths(λ_(p)) that is proportional to a unique correlation number k,${\left( \frac{a}{r} \right)\frac{\lambda_{s}^{m}}{\lambda_{p}^{n}}} = k$

such that λ_(s) can be any wavelength of sound and λ_(p) can be anywavelength of light. The variable a is a term that relates soundwavelengths to an octave system and r is a number that represents theperceived brightness of light. Powers of the ratio m and n are numbersbetween 0.2≦m≦2and 0.2≦n≦2.

As another aspect of this invention, k is the only consistentcorrelation number that is unique among all wavelength combinations ofnote sounds and colors and reveals a continuous, seamless, and mutualcorrelation between light and sound. The correlation number k is relatedto a ratio of light and sound velocities.

This method eliminates imposing any harmony that is based on humanexperience or choice, and thus brings complete universality andobjectivity. Hence, the method based on this equation eliminates allconcerns regarding any need for subjective inputs, references,selections, and user-based options.

In an aspect of this invention, a is correlated to a number O′ whichrelates any sound to an octave:${a = {0.5 \times 2^{(\frac{O + b}{\sqrt{\phi}})}}},$

Here, φ is the Golden Ratio and O′ is given by:${O = {{int}\left( {{\log\left( \frac{d}{\lambda_{s}} \right)}/e} \right)}},$

In the above equations, b, d, and e are numbers between −5≦b≦5,0.05≦d≦40, and 0.1≦e≦1.

In an aspect of this invention, r is correlated to O′ as given:${r = 2^{({\frac{O + b}{2\sqrt{\pi}} + c})}},$

Where c is a number between −2≦c≦2.

Hence, the present method uniquely and seamlessly correlates any lightwavelength to a sound wavelength as:${\lambda_{p} = \frac{\lambda_{s}}{\left( \frac{2{kr}}{2^{(\frac{O + b}{\sqrt{\phi}})}} \right)^{1/m}}},$And the present method uniquely and seamlessly correlates any soundwavelength to a light wavelength as:${\lambda_{s} = {\lambda_{p} \times \left( \frac{2{kr}}{2^{(\frac{O + b}{\sqrt{\phi}})}} \right)^{1/m}}},$

Hence, in its most basic form, the present method comprises thefollowing steps:

-   -   inputting at least one wavelength, wavelength of light (λ_(p)),        r and/or wavelength of sound (λ_(s));    -   calculating the corresponding wavelength of sound and/or light        using a single equation:        ${\left( \frac{a}{r} \right)\frac{\lambda_{s}^{m}}{\lambda_{p}^{n}}} = k$    -   outputting the calculated wavelength.

In another embodiment (FIG. 1), the present method comprises:

-   -   inputting at least one wavelength, wavelength of light (λ_(p)),        r and/or wavelength of sound (λ_(s)) (101);    -   if λ_(s) is an input (101),        -   checking whether O′ is an input (102);        -   if O′ is not an input, calculating O′ from the equation            (103):            ${O = {{int}\left( {{\log\left( \frac{d}{\lambda_{s}} \right)}/e} \right)}},$        -   calculating r from O′ using the equation (104):            ${r = 2^{({\frac{O + b}{2\sqrt{\pi}} + c})}},$        -   then calculating the corresponding wavelength of light using            the equation (105):            ${\lambda_{p} = \frac{\lambda_{s}}{\left( \frac{2{kr}}{2^{(\frac{O + b}{\sqrt{\phi}})}} \right)^{1/m}}},$        -   outputting the calculated wavelength of light (λ_(p)) and r            (106).    -   if λ_(p), r are inputs (101);        -   calculating the O′ using the equation (107):            $O = {{int}\left( {{\left( {\frac{\log\quad r}{\log\quad 2} - c} \right) \times 2\sqrt{\pi}} - b} \right)}$        -   then calculating corresponding wavelength of sound using the            equation (108):            $\lambda_{s} = {\lambda_{p} \times \left( \frac{2k\quad r}{2^{(\frac{O + b}{\sqrt{\phi}})}} \right)^{1/m}}$        -   outputting calculated wavelength of sound (λ_(s)) and O′            (106).

An apparatus exploiting the present method may be any device capable ofperforming above calculations (FIG. 2). The said apparatus may bemechanical, electromechanical, electric, electronic, analog, digital,and/or hybrid. Examples may be an electronic card, a microprocessor, acomputer, etc.

The inputs to the apparatus may be realized through any interface, suchas a keyboard, keypad, a mouse, a device measuring wavelength of soundand/or light, a touch-screen, a graphics interface, sensor, light andsound transducer measurement device, etc.

The outputs from the apparatus may be in the form of a sound and/orlight generator and/or graphics generator, e.g. full-light spectrumlamp(s), an LCD screen, monitor, TV, loudspeakers, electronic piano,keyboard, etc. and any other suitable device.

Various embodiments of the present method and apparatus are possible,only a few of them being mentioned here for the sake of illustration.

In one embodiment of the invention (FIGS. 3 and 4), the method and/orapparatus is utilized for producing a 3-D color sound correlationdisplay apparatus for referral, illustrative and educational purposes. Abasic diagram of the 3-D color and sound correlation display apparatusis constructed according to the present method, showing the entireaudible range superimposed on several layers of the visible light gamut.In FIG. 3, the contour of each layer denotes the wavelengths of thevisible spectrum. A layer is constructed for each octave with thecorresponding perceived brightness and then these layers aresuperimposed in terms of various relationships of the correlation toproduce the final diagram (FIG. 4). All wavelengths of audible sound canbe displayed or marked on the correlating wavelength of the visiblespectrum. Hence, each point represents a continuous light and soundcorrelation. To illustrate, in FIG. 4 points corresponding to the note Ffor eight octaves are marked (F0, . . . , F7). These points form an axisfor notes F in all octaves. The change of perceived brightness of thecolor correlating to different octaves of this note is obvious in thediagram. Axis for any note sounds and for any another conceivable subnotes exists as well. Once a user points to a color on the diagram, thecorrelating sound is generated and displayed. The reverse is alsopossible such that a sound is pointed or input and the correlating coloris located and displayed. The display apparatus can be in any physicalor electronic form or format and any size with at least two-dimensionaldisplay. The sequencing of the color and sound correlation involved inthis diagram is similar to the natural perception sequence in the humanbrain. This embodiment is uniquely useful for music education andillustrates the coupling of human sound and light cognition. Thecontinuous correlation is especially useful to display on stringedinstruments without discrete keys, such as violin. In anotherembodiment, the continuous light and sound correlation may be displayedby this apparatus attached to any music instrument.

FIG. 5 shows another embodiment of the invention. In this embodiment,the method is utilized for the manufacture of a color and soundcorrelating slide rule apparatus for referral, illustration, animation,and education purposes. This apparatus creates an effective medium forcombining visual and/or auditory senses with hand movements. The colorand sound correlating slide rule apparatus (50) comprises at least astationary part (51) on which at least one of the light and soundcorrelation variables, i.e. colors or notes, are labeled and/orgenerated and at least one movable part (56) which moves relative to thestationary part(s) on which at least one of the other variables arelabeled and/or generated and at least one correlating variable isdisplayed and/or generated. These parts may be in the form of anythree-dimensional object, i.e. disc, plate, strip, cube, cylinder,sphere, etc. Moreover, the relative motion between the stationary andmoving parts can be linear and/or rotational. In the preferredembodiment of the invention, both parts (51 and 56) are in the form ofconcentric circular disks, placed such that the moving part rotates onthe stationary part. For easy rotating, there is provided at least onethumb lever (54) on the rotating part (56). In addition, for precisealignment, there is a pointer (52) on the stationary part. When therotating part is aligned with the pointer on the stationary part, thecorrelating variable can be seen at a reference point. In the preferredembodiment of the invention, the reference point is a window (55) on therotating part. This window (55) shows the color correlating to themusical note on the rotating part (56), which is aligned in front of thepointer. The reverse is also possible, i.e. aligning the window (55) toa desired color, so that the pointer (52) shows the correlating musicalnote. The color and sound correlating slide rule apparatus can berotary, sliding, cylindrical, cubical, or in any other form, and mayinclude sound and/or light output as another form of displaying soundand color correlation.

In one embodiment of the invention, a new CMIDI (Color MusicalInstrument Digital Interface) file generating apparatus establishes adynamic coupling among sound and light data (FIG. 6). This apparatus(60) receives an array of instrument and sound data per any timeinterval (61). Next, it is checked whether the sound data includeswavelength of sound (62). If this information is absent, wavelength ofsound is calculated from present information (63). Each wavelength ofsound is input into the light and sound correlation apparatus (64),which in turn calculates wavelength of light and perceived brightness,r. This information is arranged in the form of a data array, preferablya CMIDI file (65). CMIDI file is an array of data in any form, format,and content including color and sound data simultaneously. In thepreferred embodiment, the sound data and correlated colors for multipleinstruments or voices per time increment are recorded in separatecolumns. This data can be output to any storage or retrieval, printing,transmitting, display or animation device, depending on the desired formof output (66).

In another embodiment (FIGS. 7-12), the present invention is utilized toprovide a visual orchestration apparatus where the sounds produced byinstruments are dynamically displayed with colors on a respectivelayout. This layout has at least two dimensions and maps at least onemusic instrument, voice, (such as human voice), and/or sound sources.The present invention, which consists of a continuous correlationbetween sound and light, can colorize all sounds produced by any musicinstrument. These include sounds of western and non-western instruments,and the sounds produced are not limited to any discrete notation system.Furthermore, sounds from any source(s) like in nature can be mapped on alayout similar to an instrument layout and these sounds can be colorizedand displayed on that layout. Hence, a visual orchestration of anycombination of instruments or sounds sources are possible. This visualorchestration apparatus (80) is an apparatus comprising functions as ameans of:

-   -   digitally defining a given or arranged instrument layout        depending on the performance, composition, or sound source(s)        (85) (FIG. 7);    -   developing a CMIDI file or data array compiled from a given        performance, composition or sound source(s) (81) using the above        explained CMIDI file generating apparatus (82), wherein the data        array includes spatial information of instruments or sound        sources on the layout (FIG. 9);    -   transferring generated CMIDI file to the instrument layout (83)        (FIG. 10);    -   outputting the orchestration in the form of frames sequenced for        each time increment wherein the colors of each sound are        displayed (84) (FIG. 11).

To illustrate, the main orchestral sections on a standard orchestrallayout are digitally identified with numbers 1 to 7 (FIG. 7) in thefirst step. Each orchestral section may also have subsections.Therefore, each instrument X is identified by a three number system X(j, q, w) where j is the section number, q is the subsection number, andw is the instrument number in that section. In the next step, the CMIDIfile generating apparatus is used with the spatial information of theinstruments defined by (j, q, w). The CMIDI file (FIG. 9) includes butis not limited to every note (Y) played by every orchestral instrument(X) for every time interval (s) correlated to the corresponding lightwavelength (λ_(p)) and its distinct perceived color brightness (r). Inthe preferred embodiment, a measure of half step intervals from a knownfrequency (N) is calculated from the note and octave and provides thenecessary information to calculate the wavelength of sound. In thefollowing step, the generated CMIDI file is transferred to theinstrumental layout for each time interval, wherein each correlatedcolor is mapped on the instrument defined by the spatial information(FIG. 10 and 11). The magnitude or shape of the mapped colored area orcolored surface on each instrument may vary in synchrony with the soundamplitude or waveform. A display frame is produced for each timeinterval of the colorized performance or composition and outputted toany display device. The output is generated on a real-time basis orstored for future display in any media format. The output of the visualorchestration apparatus provides a coupling of visual perception withmusical sounds. The output may be used to recognize and locate soundswith color coupling, identify patterns in music or sounds with colors,treat tonal deafness, and train for perfect pitch etc.

In still another embodiment (FIG. 12), the visual orchestrationembodiment is reverse engineered, so that it becomes an art composerapparatus (120). In this case, it becomes possible to compose music fora given artwork, photograph, image, piece of art etc. (121) by using thepresent invention. For this purpose, the artwork to be composed isdiscretized into small surface area fragments (grids) (122). Thediscretization grid has at least two dimensions. One dimensioncorresponds to the time frame. The other dimensions correspond to theinstrument layout in the orchestra, which plays that art. For each gridsurface area, color information is calculated (123). Then thisinformation is input into the light and sound correlation apparatus(124) which in turn outputs the corresponding wavelength of sound for agiven grid. Then the sound data is output in any suitable file formatsuch as MIDI, mp3, etc. (127). If visual orchestration is also requested(125), wavelength of sounds is input into the visual orchestrationapparatus (126). The time frame of orchestration is based on the timedimension of the image. On the other hand, the decision on whichinstrument to play the note corresponding to which grid is made based onthe instrument layout dimension of the image. Hence the orchestra playsfirst a row (or column) of an image then the next rows. In each row,every grid area is assigned to a predetermined orchestral instrument.Thus in every time frame, every instrument plays the note correspondingto the color of the grid area of its column. Of course, within thisscope, more complicated grid generation techniques and instrumentassignments may be realized, such that instruments are assigned using adepth proportion between an instrument layout and an image in threedimensions. In this manner, it will become possible to combine the depthof sight in vision with depth of sound in hearing. The generated soundscan also be used by the visually impaired to hear images with sounds.

In still another embodiment of the invention, an apparatus, whichproduces sound data incorporating three-dimensional spatial informationof an object, is developed. The spatial information is derived from thefact that the shades of colors change with depth. As the user, moves thepointer (131) across the image (132) of the three-dimensional object,the wavelength of light is calculated at every point of the displayedimage and this data is simultaneously input into the light and soundcorrelation apparatus. The light and sound correlation apparatusgenerates sound data. This sound data is played using any soundgeneration apparatus (133) such as loudspeakers, etc. This apparatus isespecially designed for individuals with William's Syndrome who haveexceptional music ability but poor perception of depth. The soundgenerated gives a virtual perception of the depth of the image. It ispossible to apply this embodiment in the form of a physicalthree-dimensional object, which is covered with a network of color-codedpressure sensors. As the user touches the object and moves his finger onthe object, a correlating sound is generated to inform him/her about thedepth of the object. This object may as well be an elastic blanket to bewrapped on various other objects the user might like to recognize by thelight and sound correlation.

In another aspect of the invention, an apparatus is developed to providea color language for the hearing impaired (FIG. 14). This apparatuscouples any AV signal with colorized sounds to aid sound perception withvisual perception. To illustrate, the hearing impaired can visuallyfollow sounds in a music performance on a TV screen, on which a picturein picture box dynamically displays colorized instrument layoutsimultaneously with the main broadcast. This invention is in analogywith sign language used for news broadcast. The color language apparatusreceives live or recorded AV input in any format (141) from twoidentical channels (142). One signal is used to arrange the instrumentlayout (143) and that signal is forwarded to the visual orchestrationapparatus (144). The visual orchestration output in this channel is sentas a second AV input to any AV apparatus with picture in picturecapability (145). The second input signal is forwarded ‘as is’ to thesame AV apparatus where this original signal is displayed as the main AVpicture, whereas the visual orchestration output signal issimultaneously displayed in the picture in picture AV frame (145).

From the descriptions above, a number of additional advantages becomeevident:

The accuracy and precision of the present method and apparatus arecompatible with the the auditory and visual sensitivity of humans thanksto the continuous nature of the disclosed function, which provides aseamless and continous correlation between light and sound. Based on thecontinous color and sound correlation disclosed, the natural correlationin the human brain has been recognized and the correlation between depthof sounds and the depth of vision provides a superior combination ofauditory and visual senses in the visual orchestration apparatus.

The present method and apparatus gives a one-to-one relationship to allsound and light wavelengths. In terms of music, this continuous functionmeans a hypothetical piano with infinitesimally small half steps andadditional octaves. The present method and apparatus can be applied tostring instruments, such that the continous range of note sounds arerepresented with a continous light and sound correlation. Based on thecontinous color and sound correlation disclosed, a music compositionwritten in one music system can be transferred to another music systemsuch as from monophonic to polyphonic or vice versa.

In addition, the present method and apparatus are applicable at anyambient conditions with the same accuracy and precision because k is anumber which can be proportionated to sound velocity. This is especiallyimportant in colorization of orchestration, because exact performance ofinstruments depend on ambient conditions. This method can accommodateand compensate changes in instrument performance and the medium.Provided that the correlation is based on wavelenghts, this method alsoeliminates the effect of the transmitting medium between the points ofsource and perception.

To summarize, the present invention discloses a direct and uniqueproportion between the wavelengths of visible light and audible soundthat is seamless and continuous. The main strength of the presentinvention lies in the fact that it provides a universal correlationbetween wavelengths of sound and light. It can be applied to allwavelengths of light and sound, without any user intervention ordependency. Hence, it effectively utilizes the human color and soundcognitions without bias. This invention provides a universal method andapparatus capable of mutually correlating sound and light wavesseamlessly, which satisfies long-felt human needs. As such, it will havemany implications in science, engineering, medicine, art, music,education, and aiding the impaired. According to Harvard Dictionary ofMusic, “the physical and psychological relationship between colors andsound seems to be existent but quite difficult to obtain andcharacterize.” Finally, this invention has achieved the impossible.

While my above description contains many specifities, these should notbe construed as limitations to the scope of the invention, but rather asan exemplification of various embodiments thereof. Many other variationsare possible. Accordingly, the scope of the invention should bedetermined not by the embodiments illustrated, but by the appendedclaims and their legal equivalents.

1. A universal method of establishing a continuous, mutual correlationbetween sound and light, comprising the steps of: inputting at least onewavelength, wavelength of light (λ_(p)), r and/or wavelength of sound(λ_(s)); calculating the corresponding wavelength of sound and/or lightusing the equation:${\left( \frac{a}{r} \right)\frac{\lambda_{s}^{m}}{\lambda_{p}^{n}}} = k$outputting the calculated wavelength; wherein, k is the only consistentcorrelation number that is unique among all wavelength combinations ofnote sounds and colors and reveals a continuous, seamless, and mutualcorrelation between light and sound, and related to a ratio of light andsound velocities, a is a term that relates sound wavelengths to anoctave system, r is a number that represents the perceived brightness oflight, and m and n are powers between 0.2≦m≦2 and 0.2≦n≦2.
 2. The methodof claim 1, if λ_(s) is an input, further comprising the steps of:checking whether O′ is an input (102); if O′ is not an input,calculating O′ from the equation (103):$O^{\prime} = {{int}\left( {{\log\left( \frac{d}{\lambda_{s}} \right)}/e} \right)}$calculating r from O′ using the equation (104):$r = 2^{({\frac{O^{\prime} + b}{2\sqrt{\pi}} + c})}$ then calculatingthe corresponding wavelength of light using the equation (105):$\lambda_{p} = \frac{\lambda_{s}}{\left( \frac{2k\quad r}{2^{(\frac{O^{\prime} + b}{2\sqrt{\phi}})}} \right)^{1/m}}$outputting the calculated wavelength of light (λ_(p)), r (106); whereinthe b, c, d, and e are numbers −5≦b≦5, −2≦c≦2, 0.05≦d≦40, 0.1≦e≦1, and

is the Golden Ratio.
 3. The method of claim 1 or 2, if λ_(p), r areinputs, further comprising the steps of: calculating O′ using theequation (107):$O^{\prime} = {{int}\left( {{\left( {\frac{\log\quad r}{\log\quad 2} - c} \right) \times 2\sqrt{\pi}} - b} \right)}$then calculating the corresponding wavelength of sound using theequation (108):$\lambda_{s} = {\lambda_{p} \times \left( \frac{2k\quad r}{2^{(\frac{O^{\prime} + b}{\sqrt{\phi}})}} \right)^{1/m}}$outputting the calculated wavelength of sound (λ_(s)) and O′ (106).
 4. Amutual sound and light correlation apparatus exploiting said methodaccording to any one of claims 1 to 3, characterized in that it is adevice capable of performing said calculations of the method, andoutputs the wavelength of sound and/or light when any wavelength oflight and/or sound is input along with other relevant data.
 5. Anapparatus according to claim 4, characterized in that the inputs arerealized through an input interface comprising a keyboard, a keypad, amouse, a device measuring wavelength of sound and/or light, atouch-screen, a graphics interface, sensor, measurement device etc. 6.An apparatus according to claim 4 and 5, characterized in that theoutputs are realized through an output interface comprising any soundand/or light generator and/or graphics generator.
 7. An apparatusaccording to claims 4 to 6, in the form of a continuous color and soundcorrelation display apparatus, wherein all wavelengths in the audiblesound range are correlated with light wavelengths on several layers ofthe visible light gamut such that a layer is constructed for each octavewith corresponding perceived brightness, and then these layers aresuperimposed, on which a user may point physically or electronically toa color to hear and/or read the correlating sound, or may pointphysically or electronically to a sound to see and/or read thecorrelating color, wherein the color and sound correlation is displayedin at least two dimensions.
 8. An apparatus according to claim 7,characterized in that said continuous light and sound correlation isdisplayed by this apparatus attached to any music instrument.
 9. Anapparatus according to claims 4 to 6, in the form of a color and soundcorrelation slide rule apparatus (50), which comprises at least astationary part (51) on which at least one variable of the light andsound correlation, i.e. colors or notes, is labeled and/or generated andat least a movable part (56), which moves relative to the stationarypart(s) on which at least one of the other variables is labeled and/orgenerated and at least one correlating variable is displayed.
 10. Anapparatus according to claim 9, characterized in that said parts are inthe form of any three-dimensional object wherein the relative motionbetween parts is linear and/or rotational.
 11. An apparatus according toclaim 9 or 10, characterized in that parts (51 and 56) are in the formof concentric circular disks, placed such that the moving part rotateson the stationary part and that it comprises at least one thumb lever(54) on the rotating part (56), and a pointer (52) on the stationarypart, such that, when the rotating part is aligned with the pointer onthe stationary part, the corresponding variable can be seen at areference point.
 12. An apparatus according to claim 11, characterizedin that said reference point is a window (55) on the rotating part,which shows the color correlating to the musical note on the rotatingpart (56) in front of the pointer.
 13. An apparatus according to claims4 to 6, in the form of a CMIDI (Color Musical Instrument DigitalInterface) file generating apparatus, which receives an array ofinstrument and sound information per any time interval (61), next,checks whether the sound information includes wavelength of sound (62),then if said information is absent, calculates wavelength of sound fromthe present information (63), then inputs each wavelength of sound intothe light and sound correlation apparatus (64), which in turn calculateswavelength of light and perceived brightness, r, which information isarranged in the form of a data array, preferably a CMIDI file (65) whichis in turn transferred to any storage or retrieval, printing,transmitting, display or animation device, depending on the desired formof output (66).
 14. An apparatus according to claims 4 to 6, in the formof a visual orchestration apparatus (80) comprising functions as a meansof: digitally defining a given or arranged instrument layout dependingon instruments in a performance, composition, or sound sources (85);developing a CMIDI file or data array compiled from a given performance,composition, or sound sources (81) using the above explained CMIDI filegenerating apparatus (82), wherein the data array includes spatialinformation of instruments or sound sources on the layout; transferringthe generated CMIDI file to the instrument layout (83); outputting theorchestration in the form of frames sequenced for each time incrementwherein the colors of each sound are displayed (84).
 15. An apparatusaccording to claims 4 to 6, for composing music for a given artwork,photograph, image, piece of art etc. (121), which discretizes said imageinto small surface area fragments (grids) (122) such that thediscretization grid has at least two dimensions wherein one dimensioncorresponds to the time frame and the other dimensions correspond to theinstrument layout in the orchestra, then for each grid surface area,calculates color information (123), then inputs this information intothe light sound correlation apparatus (124) which in turn outputs thecorresponding wavelength of sound for a given grid, and outputs thesound data in any suitable file format such as MIDI, mp3, etc. (127).16. An apparatus according to claim 15, characterized in that if visualorchestration is also requested (125), wavelength of sounds is inputinto the visual orchestration apparatus (126).
 17. An apparatusaccording to claims 4 to 6, which derives spatial information accordingto the shades of colors that change with depth, and as the user, movesthe pointer (131) across the image (132) of the three dimensionalobject, calculates the wavelength of light at every point of the image,simultaneously inputs this data into the light sound correlationapparatus, and plays the generated sound data using any sound generationapparatus (133) such as loudspeakers, etc.
 18. An apparatus according toclaim 17, characterized in that it is in the form of a physicalthree-dimensional object, which is covered with a network of color codedpressure sensors, such that as the user touches the object and moves hisfinger on the object, a correlating sound is generated to inform him/herabout the depth of the object.
 19. An apparatus according to claim 18,characterized in that it is in the form of an elastic blanket to bewrapped on various other objects the user might like to recognize by thelight and sound correlation.
 20. An apparatus according to claims 4 to6, characterized in that it receives live or recorded AV input in anyformat (141) from two identical channels (142), uses one signal toarrange the instrument layout (143) and forwards that signal to thevisual orchestration apparatus (144), wherein the visual orchestrationoutput in this channel is sent as a second AV input to any AV apparatuswith picture in picture capability (145) and the second input signal isforwarded ‘as is’ to the same AV apparatus where this original signal isdisplayed as the main AV picture, whereas the colorized AV signal issimultaneously displayed in the picture in picture AV frame (145).